Configurable AI model selection with automatic fallback (fixes #91)#102
Configurable AI model selection with automatic fallback (fixes #91)#102tbartik wants to merge 1 commit into
Conversation
…crosoft#91) The extension's AI features previously selected a single Copilot chat model by family (gpt-5.4-mini -> ... -> gpt-4.1). When that model is disabled for the user's Copilot plan or organization it stays *selectable* but returns an empty response, so JSON parsing fails and AI features error out (or silently produce nothing) with no actionable guidance. This adds a central model-selection module and routes every AI feature (panel LLM helpers and the rule compiler) through it: - core/llm-models.ts: single source of truth for the preferred model (new aiEngineerCoach.preferredModel setting + in-memory runtime override). Auto mode orders candidates by a vendor-neutral capability heuristic and iterates them, transparently skipping any model that returns an empty response. Pure helpers (scoreFamily, orderModelsByPreference, dedupeModelsById) are unit tested. - panel-llm.ts: callLlm/callLlmJson iterate candidate models and skip empty responses; when every model returns empty they throw a clear message that points the user at the new "AI Model" picker. Upstream JSON-repair and structured-output handling are preserved. - Dashboard sidebar gains an "AI Model" dropdown bound to the setting via new listModels/setModel RPCs. - rule-compiler.ts uses the shared candidate list instead of a hardcoded gpt-4.1 family.
|
Heads-up on overlap with #98 ("add Gemini CLI support and provider-agnostic AI settings"). Both PRs add a user-configurable preferred-model setting + a model picker, so they touch some of the same files ( This PR is specifically the fix for #91. #98 refactors model selection (preferred id + a hardcoded high-capability family list) but still does a single Possible conflict: the setting key differs — #98 uses I'm glad to rebase this on top of #98 (layering the empty-response fallback onto their selection logic) or vice-versa — whatever ordering is easiest to review. The Gemini CLI parser in #98 is independent and valuable on its own. |
@microsoft-github-policy-service agree |
There was a problem hiding this comment.
Pull request overview
This PR makes the extension’s Copilot chat model selection configurable and resilient by centralizing model selection, adding a dashboard “AI Model” picker, and iterating through candidate models to avoid failures caused by models that are selectable but return empty responses (e.g., disabled by plan/org).
Changes:
- Introduces
src/core/llm-models.tsas a shared model-selection layer (preference + auto ordering + helpers + tests). - Updates LLM callers to iterate candidate models and skip empty responses; adds sidebar UI + RPC methods to list/select models.
- Switches the natural-language rule compiler to use the shared candidate model list instead of a hardcoded family.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/webview/panel-request-service.ts | Adds RPC handlers (listModels, setModel) used by the dashboard model picker. |
| src/webview/panel-llm.ts | Updates callLlm/callLlmJson to try multiple models and handle empty responses; re-exports model APIs. |
| src/webview/panel-html.ts | Adds “AI Model” dropdown to the dashboard sidebar. |
| src/webview/app.ts | Populates the model dropdown via RPC and persists selection changes. |
| src/core/types/rpc-types.ts | Extends RPC type map with listModels / setModel. |
| src/core/rule-compiler.ts | Uses shared candidate model list and falls through on empty responses. |
| src/core/llm-models.ts | New centralized model selection + preference persistence + ordering/dedupe helpers. |
| src/core/llm-models.test.ts | Adds unit tests for scoring, ordering, and deduplication helpers. |
| package.json | Adds aiEngineerCoach.preferredModel setting with markdown description. |
| CHANGELOG.md | Documents the new configurable model selection and fallback behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const all = await vscode.lm.selectChatModels({}); | ||
| if (all.length === 0) { | ||
| throw new Error('No language model available. Make sure GitHub Copilot is installed and signed in.'); | ||
| } | ||
| const preferred = getPreferredModelId(); | ||
| const ordered = orderModelsByPreference(all, preferred); | ||
| runtimeDebug('llm', 'models-available', | ||
| `preferred=${preferred ?? '(auto)'} count=${ordered.length} ` + | ||
| `models=${ordered.map(m => `${m.id}(${m.family})`).join(',')}`); | ||
| return ordered; |
| const response = await model.sendRequest(retryMessages, options, cts.token); | ||
| for await (const chunk of response.text) text += chunk; |
| // Iterate candidates so a model that is disabled for the plan/organization | ||
| // (which returns an empty response) transparently falls through to the next. | ||
| for (const model of candidates) { | ||
| const response = await model.sendRequest(messages, {}); | ||
| let result = ''; | ||
| for await (const chunk of response.text) { | ||
| result += chunk; | ||
| } | ||
| if (result.trim().length === 0) continue; | ||
|
|
||
| // Extract markdown from code block if wrapped | ||
| const fenced = result.match(/```(?:markdown)?\s*\n([\s\S]*?)```/); | ||
| if (fenced) return fenced[1].trim(); | ||
| if (result.includes('---')) return result.trim(); | ||
| // Extract markdown from code block if wrapped | ||
| const fenced = result.match(/```(?:markdown)?\s*\n([\s\S]*?)```/); | ||
| if (fenced) return fenced[1].trim(); | ||
| if (result.includes('---')) return result.trim(); | ||
| return null; | ||
| } |
| </div> | ||
| <div class="sidebar-filter"> | ||
| <label for="model-filter">AI Model</label> | ||
| <select id="model-filter"><option value="">Auto (first available)</option></select> |
|
How about we try |
Description
The extension's AI features (skill generation, quizzes, code reviews, did-you-know, rule compilation, etc.) previously selected a single Copilot chat model by family (
gpt-5.4-mini→gpt-5-mini→gpt-4.1-mini→gpt-4.1, then first-available). When that model is disabled for the user's Copilot plan or organization, it is still selectable viavscode.lm.selectChatModelsbut returns an empty response. The empty text then fails JSON parsing, so AI features error out — or silently produce nothing — with no actionable guidance for the user.This PR makes model selection configurable and resilient:
src/core/llm-models.ts— a single source of truth for model selection:aiEngineerCoach.preferredModelsetting plus an in-memory runtime override (so a change applies to the next request immediately and persists across reloads).scoreFamily) and iterates them, transparently skipping any model that returns an empty response instead of relying on a single pick.scoreFamily,orderModelsByPreference,dedupeModelsById) are unit-tested.panel-llm.ts—callLlm/callLlmJsoniterate candidate models and skip empty responses. When every model returns empty, they throw a clear message pointing the user at the new "AI Model" picker. Existing JSON-repair (balanceTruncatedJson) and structured-output fallback behaviour are preserved.listModels/setModelRPCs (panel-html.ts,app.ts,panel-request-service.ts,rpc-types.ts).rule-compiler.ts— uses the shared candidate list instead of a hardcodedgpt-4.1family, so rule compilation benefits from the same fallback.The capability ordering is deliberately organization-neutral: it does not assume which models any particular plan has enabled — a disabled model simply returns empty and the loop falls through to the next candidate.
Related Issues
Fixes #91
Checklist
npm run checkpasses (typecheck + lint + spellcheck + knip + tests)src/core/parser-vscode.test.ts›findVsCodeDirs) fails on a cleanmaincheckout as well; it is unrelated to this change.src/core/llm-models.test.ts(11 cases for scoring, ordering, and de-duplication)markdownDescriptioninpackage.json